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ABSTRACT 

The purpose of this study iras to determine the 
relative importance of six criteria of project success and to develop 
a composite criterion of educational project ;&^ccess* Criteria 
considered vere schedule, cost, quality/performance, folloir-onvorlc, 
spin*of benefits, customer/client satisfaction, and over^all project 
success* Tiro hundred and six people associated irith a sample of 146 
educational projects fj,r$t ranked the criteria and then rated their 
oirn projects accordingly* A number of diagrams graphically present 
the relative importance of the various criteria as perceived by 
different groups of respondents. Findings of the study shoired that^ 
achieving pjioject objectives vas considered most important and 
meeting time and cost constraints vas considered rather unimportant 
in determining the .success' of projects. A copy*of the survey 
instrument used to gather data^for study is included in the^ 
appendix* <Author/JG) 
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nU: DrM:LOPMFiir OF SUCCiiSS CRI'reRIA FOR liDUailOMAL 
RHSFJU?ai AND DirVFLOWrr PROJECTS 



Rodney J, Ball* 
Desmond L, Cook+ 



ABSTRACT 



The purpose of the study was to determine the relative inqportance 
of six criteria of project success and to develop a cCTi?)osite criterion 
of educational project success, TVro hundred and six people associated, 
with a san?)le of one hundred forty-:Six educational projects first 
ranked the criteria and then rated their projects accor'l ngly. Achieving 
project objectives was considered most iirportant, while meeting project 
time schedule and cost constraints were rather unijnportant in uetemining 
the succe^ss of ^'projects. This study represents a significant step in 
developing criteria for the determination of the success of educational 
projects, 

INTRODUCTION 

The determination of the final success or failure of research and 
development projects is ySn iiiqportant problem for those conceirned with 
project managenient in Almost all fields ^f endeavor. One ijni^iate 
problan encountered in determining project sucxess is defitiing v^at is 
meant by "success/* If seme criterion can be established, research can 
be directed toward discovering which variables associated with the 
management of projects are critical in determining a successful project 
as contrasted with those inherent in the effort. Project managers and 
their staffs can t\\^ concentrate on those variables as they work to 
ensure the success of their project efforts. 

Previous research at Harvard University?* and the Massachusetts 
Institute of Technology^ has indicated that several criteria are considered 
v^en the success of various weapons systems oriented projects are rated. 
It can not be asstaned, however, that these success criteria have the 
same relative irnx)rtance in^education. For exaii5)le, in weapon systen 
development the performance criterion proved to be by far the most im- 
portant- In education, where performance is often quite difficult to 
measure, remaing within cost and time constraints may be as in^xjrtant 
or more important than meeting performance specifications. 



* Research for Better Schools, Inc, Philadelphia, Pennsylvania 
■*■ Educational Program Management Center, The Ohio State University, 
Columbus* Ohio,, 
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Cook recently conducted a research study which resulted in a conceptual 
framework to be used for the study of project managanent in education. 
This work was reported at the Third Annual Seminar Syn^sium of the 
Project Management Institute in 1971. The purpose ot this effort was to 
develop a framework which could be used to , integrate the results of past 
and current studies and to direct further research on project management, 
A schanatic of this conceptual framework is presented as Figure 1, Project 
success represents a majof component of the conceptualization. It includes 
such potential criteria as meeting schedule dates, rejnaining within cost 
constraints, meeting quality specifications (objectives) , satisfyiiig the 
customer, obtaining spin-off benefits, and securing stpport for additional 
Cfollow-on) efforts. 

jThe conceptual framework suggests that different persons may have 
different viewpoints on project success and that these would be reflected 
in their ranking of possible success criteria. For exan^le, project 
managers may be most concerned with time, cost, and performance, ^ereas, 
parent organization representatives may be quite concerned about spin-off 
benefits, follow-on work, and custftmer or funding agency satisfaction, 

PURPOSE AND HmriHESES 

The general purpose of this study was to explore some of the hypotheses 
suggested by Cook's conceptual framework in regard to the relative ijn- 
portance of different possible criteria of project success in education. 
Specifically, the objectives of the study were; 

CA) To rank schedule, cost, quality Cperfoimance) , follow-on work, 
spin-off, and customer satisfaction as possible criteria of 
educational project success, 

CB) To quantify a con^xjsite criterion of education project success 

, enployiJig the sub-criteria of schedule, cost, quality, follow-on 
work, spin-off, and customer satisfaction. 

The final ranking of the several criteria of project success has 
implications for the many governmental and non- governmental organizations 
who must determine the ultimate success of the educational projects they 
fund, monitor, and/or plan to ijt(>laiient or institutionalize. That is, 
the determination of what constitutes a successful project is a necessary 
initial step in the evaluation of projects- 

Once it is known ^at characterizes a successful project, research 
cart be directed toward discovering ^ich variables in the management o£ 
projects are critical in detexmining a successful project. If the conposite 
criterion of project success ^resulting from this study is used as the 
dependent variable, research can be conducted on the relationship of 
selected independent variables of the framework in order to more firmly 
establish those n%nagement actions on a project that will Titake it successful- 
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Within the general purpose and specific objectives, the study 
proceeded with the following hypotheses- 

1) Project principal investigators project managers), parent 
organizatim representatives, and government monitors view 
the relative in?)ortance of the criteria differently. 

2) Project principal investigators with previous project 
management experience view the criteria differently than 
those without prior experience. 

3) Project principal investigators of sponsored projects 

^ Cconducted under the supervision of a university fdculty 

menfcer) view the criteria differently than those conducting 
non-spcmsored projects* 

4) Project principal investigators and parent organizatim 
representatives at large institutions view the criteria 
differently than those from small institutions. 

5) The correlations of the different criteria with ratings 
of overall project success reflect the ranked relative 
importance given the criteria, 

PRbCEDURES 

A san^le of educational projects was first selected and then the 
personnel associated with the conduct and monitoring of the^projects 
were identified. Hiese people represent the source of data for the 
study. The sanple^ data collection instrunents, and analysis of data 
tjrocedures are specified in detail below. 

\ " 
Sanple Wid Data Gbllection 

The data for the study were obtained by means of a ranking form 
and a rating form administered to the project manager, parent organizatim 
representative, and government project monitor associated with each 
project in a saiqple of 146 terminated projects funded in one of the several 
Regions under the Regional Research Program of the National Center for 
Educational Research and Development, United States Office of Educatim. 
The projects were all operated and terminated during the period from 
September, 1967 to January, 1972; had a maxinnin government sij^jport of 
$10,000, and were usually conducted or managed by a sii>gle '*principal 
investigator,** 
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For the purpose of this study, the six possible criteria were defined 
given below, - \^ 

Schedule refers to meeting project milest^ae deadlines and the 
schetluled project termination date as established in the project 
contract. Success on thi,s dimension means thiit the project was 
completed on or before the schedul^ teminatioiVdate and that all 
subsections or coiiq)onents of the project were conqsdeted as scheduled. 

Cost refers to meeting the total dollar expenditure pqredicted for 
tile entire life of the project. Success of this dimerts^ion jneans 
that the actual dollar cost of the project was less than^or coincided 
with the predicted cost, \ 

\ ^ 

CXiality/Perfonnance refers to the aiiiievement of the goals ^ 
objectives ot the project. Success on this dimension means tfi^t 
the perfonnance standards specified by the project objectives wojre 
met or exceeded. 




Follow-qn Work refers to additional coritfacts or follow-on work 
obtained as a result of the success of the original project and 
consequent enhancanent of the reputation of the researcher. This 
criterion is not, however, concerned with additional work obtained 
as a logical or technical extension of the original project. It 
is concerned only with whether or not additional work is obtained 
as a result of the success of the original project. 

Spin-off refers to indirect benefits received by the parent institution, 
its faculty , and the project staff. Success on this dijnension means, 
for exanple, that the project and/or the parent institution's faculty 
developed skills as a result of working with the project which c&n 
be applied to other areas of the institution's operations. Another 
in^rtant spin-off benefit is the recognition an institution often 
gains from a highly succerisful project. 

Customer /Client Satisfaction refers to the degree to Miich the funding 
agency or user of the end product of the j^roject is satisfied. This 
critericri is not necessarily concerned with the quality of the project 
as indicated by whether or not the original objectives and perfontwSice' 
standards were met. It is concerned only with the satisfaction of the 
funding agency or customer with the project or its end product. 

O verall Success refers to a general or overall evaluation of the project 
iifort. Was it con5>leted as scheduled? Were the objectives and 
performance Standards met? Were all interested parties satisfied with 
the effoirt? Was the project conducted within the estijnated cost? Did 
the project generate additional projects? Tliis criterion is not con- 
cerned with any one specific aspect or objective of the project, Tlie 
concern is with making a general value statement indicating the degree 
to which the project was a success. 
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The respondents were requested to rank the relative in^^or- 
tance of the six criteria using a paired comparison format, / 
I;ach respondent was then asked to rate a completed project-^ ■ 
with which he had been associated on each of the six criteria 
and a rating of overall project success. These ratings were 
made on a Likert type ^scale, A copy of these instruDnents 
appears in the AppendiJC. 

The hypotheses required that data be gathered regarding 
(a) the size of the parent organisation housing the projects 
in the saiiple, (h) the previous project managem^t experience 
of the principal investigator, and (cj irfiether or not each 
project In the sample was a sponsored Cunder the siq?ervision 
of a university faculty memberj or non-spunsored project. 

Size Nearly all the projects in the sample were 
conducted at a college or university. The total degree credit 
enrollment in Fall 1969 was used as the measure of parent 
organization size, 

Hxperience Each principal investigator was asked to 
indicate the nunber of projects he had managed or directed 
prior to the project specified in the sanple. This number 
was used as a measure of previous project management experience. 

Sponsorship — Information regarding whether a project 
was sponsored or not was cfctained from the records of a regional 
office of the Regional Research Program, 

Data Analysis 

The analysis of the data was organized in terns of answering 
the questions raised by the five hypotheses of the study. In order 
to answer these questions, the data were first partitioned based 
ui>on the role of the respondent, previous experience of the 
principal investigator, whether tfr nQt the project was sponsored, 
and the size of the parent organization or institution. 

In order to obtain a ranking of th^e six criteria of project 
success, the paired comparison data were run on the PCMP 
computer program of the Psychology Department of The Ohio State 
University, This program follows the assuir^tions of the Thurstone 
Case III HKKiel and confutes a scale value for each criterion. 



In order to measure the agreement of the different partition 
groups in their rankings of the relative importance of the 
criteria, an analysis of variance procedure was used in 
estimating the reliability of scale values given the criteria 
by the grotjps. This represents the degree of agreement of the 
groups or judges in the scale values they have given the cri- 
teria- A coefficient of 1,00 represents a»H)lete agreement. 

In order to obtain a coii?)osite criterion of project 
success, the ratings of projects oh the six criteria plus 
overall success made by the respondents were run on the 
\i/heny Test Selection Program of the Psychology Department of 
The Ohio State Uhiversity, The program is a regression 
analysis program which selects, in order of decreasing^ im- 
portance, the variables contributing significant vari- 
ance to the prediction of the criterion variable (overall 
success). For the purpose of this paper, the prediction 
equations are not presented. 

The correlation coefficients of the six criteria with 
overall project success are '^iscussed in order to indicate 
the extent to which the ranked relative importance of the 
criteria is reflected in the ratings of overall project success. 



FINDINGS, CONCLUSIONS, AND LIMLIATIONS 

The results of the data analysis are interpreted in 
terms of the hypotheses of the study* These hypotheses 
and findings are listed below along with a discussion of the 
data analysis results that pertain to each. In order to 
answer the fifth hypothesis, the correlation coefficients of 
the six possible criteria with overall project success, ratings 
are discussed under each of the first four hypotheses. 

Hypothesis 1 Project principal investigators (project managers \ 
parent organizaticfi representatives, and ^government monitors 
view the relative imjortance of the criteria difftetgnt^b ^. 

The data analysis indicates that this hypothesis is false. 
The scale values given the criteria by these two groups are 
shown in Figure 2, They are nearly the sanie except for 
a reversal of schedule and cost, which are the two least 
iiH)brtant in each case. The reliability coefficioit of 
0,9559 indicates a considerable amount of agreemait between 
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the two gi'oups in ranking the criteriSi. The criteria in order 
of importaiKe are quality, customer satisfaction, spin-off, 
follow-on, ami then schedule ^d cost. 

IVlicn the principal investigators and parent organization 
representatives rated projects using the criteria, the following 
correlations with overall success ratings resulted respectively. 

o quality (0.5888 and 0.6559) 

0 custonier satisfaction (0.5748 and 0.5730) 

0 spin-off (0.3470 and 0.4746) 

o follow-on (0.3112 ancT 0.4008) 

o' schedule (o.l519 and 0.1362) 

0 cost (0.0101 and 0.0019) 

Note that the correlation coefficients decreased in size 
according to the ranked relative ii^^ortance of the criteria* 

The ^;g\ferwnent monitor for all the projects in the study 
sample ranked the importance of the criteria in the order of 
quality, spin-off, folloW"On» customer satisfaction* cost and 
schedule. The only important difference in the government monitor's 
ranking of criteria is the placement of customer satisfaction 
in a much lower position* Otherwise, the government monitor's 
rankings are basically the same as thos_e of the principal 
investigators and parent organization representatives, 

tVhen the government monitor rated projects using the 
criteria, the following correlations with overall success 
ratings resulted, 

o spin-off (0,7489) 

o quality (0,6459) 

o follow-on (0,6178) 

o customer satisfaction (0,4729) 

o cost (0,2149) 

o schedule (o,1253) 

Hie 1 ank order of criteria based upon the size of the correlation 
coefficients results in the same order for the criteria as 
ivhen previously ranked for iTH>ortance except for the reversal 
of quality and spin-off in the first two positions. 
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liypotheses 2 Project principal investigators with previous 
manaflement experience view the criteria differently tiian those 
without prior experience . 

The data analysis indicates that this hypothesis is 
false. The scale values given the criteria by these two 
ijroups are shown in Figure 3. There is one important exception, 
however, in their ranking of the T;elative importance of the criteria. 
Principal investigators without prior experience ranked follow-on 
work markedly lower than spin-off, whereas little or no difference 
was indicated between follow-on work and spin-off by experienced 
principal investigators. 

When these groins rated the stKxess of projects, follow-on 
work showed no correlation with overall success (r=0,0365), for 
in-experienced principal investigators, whereas, for experienced 
ones it correlated second highest (r=0,6039) of the six criteria 
viith overall success ratings^ It seans then t^at follow-on work 
is an important criterion to principal investigators who have 
previously managed projects but it is not important to principal 
investigators who have no^previous project management experience, 

Otheivise, these two gn>t?>s view the criteria veiy much 
the same as is indicated by the reliability coefficient of 0,9771, 
Again, quality is ranked the highest with customer satisfaction 
also hi^ly important. Cost and schedule are of least ijiportance , 

Hypotheses 3 -- Project principal investigators of sponsored 

f rojects (conducted under the superyision of a university 
acuity member] view the criteria differently than those 
conducting non-sponsored projects . 

The data analysis indicates that the hypothesis is false, 
The scale values given the criteria by the two groups are vety 
nearly the same. These values are shown in Figure 4- The rank 
order of the criteria is quality, custoatier satisfaction, spin-off, 
follow-on work, cost, and schedule. The same general pattern emerges 
as with previous comparisons. Quality is at the high end of the 
scale, the next three criteria are groijped together in the middle, 
and cost and schedule are at the bottom, Agreanent between the 
groups is high as indicated by the reliability coefficient of 0,9703, 

When principal investigators of sponsored projects (usually 
graduate students in this sample) applied the criteria to rating 
a project, customer satisfaction had the highest correlation (0,7145) 
with overall success ratings and spin-off had a relatively low correlation 
(0,2692), as did cost and schedule (0,2223 and 0,2633), 



10 



ERIC 



10 



Myixjthesis 4 — Project principal investigators and parent 
orj^onizat Lon representatives of large institutions view the 
criteria ctiffercntly tnan tnose trom smail institutions . 

"Die data muilvsis indicates that this hypotheses is false, 
the scale values given the criteria by the two groups are shavn in 
Fiijure 5, The high level of agreement is seen in the reliability 
uocf ficient of 0,9241, As before, quality was ranked the highest 
with custofier satisfaction, spin-off and follow-on work grouped 
in the middle, and cost and schedule ranked as the least in^Jortant, 

IVhen the >iroups rated the success of projects, the criterion 
of ^pin-off benefits correlated rather high 1(0,5867) with overall 
project success for respondents from lar^e-^ institutions and quite 
lou^ rO,1927) for /^^ondent:^ from small institutions. It appears 
ihcn that priTicipa] investigators and parept- organization repre- 
sentatives from large institutions take spin-off benefits more^ into 
accamt when ratinj; the success of projects than do such persons 
from small institutions. 

Conclusions 

%ie following conclusions regarding possible criteria for 
determining the success of a project in the field of education 
can be drawn from this study* 

1) The criteria of quality/performance (achievement of 
the goals and objectives of the project) ds by' far 

the most important criterion in the minds of feducators, 

2) The, criteria of customer/cl^nt satisfaction, spin-off 
benefits, and follow-orf w'ork are also iTH)ortant criteria, 

3) Th€L criteria of schedule and cost are of relatively little 
importance to educators^ ^ 

41 Project principal investigators and parent organization re- 
presentatives view the importance of the criteria in the same 
manner, 

S) f pi low-on work appears to be an important criterion to prin- 
jripal investigators who have previously managed projects but 
not to principal investigators without such previous experience, 

b) Principal investigators of projects conducted under the 

sponsorship/supervision of someone else appear to give more 
importance to the criterion of customer satisfaction than do 
principal investigators of non-sponsored projects, 

7) Principal investigators and parent organization representatives 
from large institutions appear to take spin-off benefits more 
into account when rating the success of projects than do such 
persons from small institutions. 
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[.imitations 

Some lijiiitations of the ,stiK)y which miy affect the generalizability 
of the conclui;ioiis are listed below. . 

1) Nearly all the projccti; in tlic^^study saii^ilc were conducted 
at institutions of higher learning. ^ 

21 Ml projects in the sanple were relatively small as 
indicated by a maximum government funding contribution 
of $10,000. 

3) All projects in the sample were conducted in a limited 
geographic region. 

SlIMVlARy AND RECmiENDATIONS 

Ihe project principal investigators Cproject managers), parent 
organization representatives, and government monitor tended to agree 
that quality is the most xnportant of the six potential criteria o£ 
project success suggested in Cook's conceptual framework and that 
schedule and cost are the least iniportant. When these groups rated 
the success of projects, the correlations of the six criteria with 
overall success ratings reflected the ranked relative importance of 
the" criteria. 

Further researcli in rhe area of developing project success 
criteria in education ought to include data gathered from projects 
with large staffs and budgets that are conducted^in various educ^itional 
institutions across tlie countr/. Such studies might concentrate on 
ttie quantification of some of ttie success criteria explored in this 
.study. Schedule and cost are probably the easiest to quantify and 
obtain reliable measures on. They, however, are the least iJnportant 
of the six studied )iere. Others like quality /performance, customer ■ 
satisfaction, and spin-off are most difficult to quantify and 
consequently are hajrder to obtain valid and reliable. measures on. 

An important implication of this study is that research on the 
management of educational projects should be directed at maximizing 
the achievement of project objectives or performance specifications. 

The evaluation of educational projects should be concentrated 
on such criteria as quality/performance, customer satisfaction, spin- 
off benefits and follow-on work, and less upon schedule and opt. 
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Figuf^e 5 
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Listed be!ow in pairs are the criteria of project success that were defined 
ori the previous page.. Select the criterion in each pair that you feel is 
the fiiore important of the two in judging the success of a project. Place 
on **x** before the more important criterion of project success in each pair, 
Auer you have finished^ turn the p^ige and continue. 
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Project Success Rating Scale 



PROJFCT NUhBER 
PROJECT TITLE " 



PROJECT PRINCIPAL INVESTIGATOR 



The severaf criteria of project success defined befor'e appear below in a rating scale format. Rate the project 
identified above on each of these criteria. Cross out the number on «ch scale that best approximates your 
rating of the project on that criterion. At the bottom of the page indicate your familiarity with the project. 
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Overall Success. 
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